Repository navigation
fix(streaming): refuse Iceberg partition transforms that Flink's DDL cannot create - #819
Merged
fas89 merged 1 commit intoNov 4, 2026
Conversation
…cannot create
The managed lane compiled icebergConfig.partitionSpec transforms into
CREATE TABLE ... PARTITIONED BY (days(`event_time`)), and likewise for
months, hours, bucket and truncate. Flink cannot parse that. Iceberg's
Flink DDL partitions by plain columns only, because Iceberg's transforms
are hidden partitioning. So the job failed after deploy with
'Encountered "(" at line 16'. The live lane found this; it predates the
record-shapes work (#784).
fluid validate now refuses those transforms: create the table with that
partition spec before the job runs, and leave it out of the contract.
Identity partitions compile as before. Real support needs the runner to
create the table through Iceberg's catalog API, and is tracked separately.
fas89
merged commit Nov 4, 2026
33b7bf1
into
feat/managed-flink-iceberg-streaming
22 of 27 checks passed
fas89
added a commit
that referenced
this pull request
Nov 4, 2026
The feature branch moved during review: #819 (refuse Iceberg partition transforms that Flink's DDL cannot create) was squash-merged onto a89966a as 33b7bf1. A clean merge with base a89966a. #819 and this branch's 3fa5f4c both change streaming/compiler.py and tests/streaming/test_compiler.py, in hunks that do not overlap; each side's diff is unchanged in both files.
fas89
added a commit
that referenced
this pull request
Nov 4, 2026
fas89
added a commit
that referenced
this pull request
Nov 4, 2026
docs/flink-iceberg.md came in with #784 but still described the old parameters.flink.source.schema, which #817 replaced with properties.stream and properties.mapping. Docs live in forge_docs, so the guide moves there, rewritten for #817, #809 and #819, as Agenticstiger/forge_docs#196. The README and the example now link to the published page (founder decision, 2026-11-04). That link resolves once forge_docs#196 merges after #784.
fas89
pushed a commit
that referenced
this pull request
Nov 4, 2026
Managed Flink to Iceberg streaming preview, in FLUID 0.7.7 (author: Dustlab). - Recursive record types. - Kafka record shapes declared in FLUID (properties.stream and properties.mapping). - Generated FlinkDeployment resources. - Plan, apply, verify and diff integration. - Checkpoint and catalog verification. - fluid generate flink. The operator security gate covers several things: - Allowlists for the kube context, namespace, image, service account and Secrets. - Listed hostnames for every destination the job contacts: the catalog, the token endpoint, S3, the Kafka brokers and the Schema Registry. - NAME@host pairing for catalogAuth. - A pinned-IP request guard. - Refusals: no env: refs in catalog text, an https-only OAuth2 endpoint, no SQL hints, and a contract-SQL allowlist. Hardened through stacked PRs #795, #809, #817, #819, #825 and #826, and syncs #815, #821 and #824. The guide is in forge_docs (advanced/managed-flink-iceberg). The remaining live proof is tracked on Trello.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #784 (
feat/managed-flink-iceberg-streaming), after #817.The managed lane compiled
icebergConfig.partitionSpectransforms (year, month, day, hour, bucket, truncate) intoCREATE TABLE ... PARTITIONED BY (days(col)).SQL parse failed. Encountered "(" at line 16, column 21.fluid validatenow refuses those transforms, and says to create the table with that partition spec before the job runs. Identity partitions compile as before.Real support (the runner creating the table through Iceberg's catalog API with the contract's
PartitionSpec) is tracked as a P1 card, as the founder decided on 2026-11-04: https://trello.com/c/2PJQkELlNot in this PR:
docs/flink-iceberg.mdstill lists "supported partition transforms", and docs live in forge_docs.